Skip to content

feat(setup): add provider defaults and lane status - #753

Open
salmonumbrella wants to merge 10 commits into
kenn-io:mainfrom
salmonumbrella:feat/issue-634-smart-defaults
Open

feat(setup): add provider defaults and lane status#753
salmonumbrella wants to merge 10 commits into
kenn-io:mainfrom
salmonumbrella:feat/issue-634-smart-defaults

Conversation

@salmonumbrella

@salmonumbrella salmonumbrella commented Sep 3, 2026 •
edited by wesm
Loading

Copy link
Copy Markdown
Contributor

Add msgvault setup providers and msgvault setup status so operators can configure optional retrieval and people lanes without assembling each table by hand.

Provider setup chooses defaults from available Voyage, Mistral, and OpenAI keys, with a local Ollama fallback. It shows provider disclosures before writing configuration and keeps existing models, explicit schedules, and known postures unless the corresponding posture flag overrides them. Sensitive people-sweep content requires a separate --allow-sensitive opt-in.

Visual setup requires a validated capability manifest and respects custom manifest paths. Status highlights missing credentials, backend support, and consent with follow-up commands. People-provider onboarding failure restores the setup configuration edits.

Status reports provider, model, schedule, consent, and next steps for each lane. Documentation covers both commands and the recommended configuration. Provider uploads and index builds remain separate, consent-gated operations.

Refs #634. Refs #589 #599 #611 #616 #620 #628.

@roborev-ci

roborev-ci Bot commented Sep 3, 2026

Copy link
Copy Markdown

roborev: Combined Review (0bb6944)

Verdict: Address one High-severity and two Medium-severity issues before merging.

High

  • Unauthenticated remote Ollama fallback can exfiltrate archive data — cmd/msgvault/cmd/setup_providers.go:443-454 (specifically 443-450)
    When no hosted key is present, setup may select any reachable Ollama server, including a non-loopback endpoint, for text embeddings without authentication or disclosure. Archived message text could therefore be sent to a remote server. Require a loopback endpoint or explicit authenticated remote configuration and disclosure.

Medium

  • Document text is omitted from the OpenAI consent disclosure — cmd/msgvault/cmd/setup_providers.go:689-691
    When documents are enabled, document_text is added to the people-sweep inference profile, but the disclosure mentions only conversation and meeting text. Include document text in the disclosure.

  • Embedding-only Ollama model may be selected for chat — cmd/msgvault/cmd/setup_providers.go:627-639
    If only nomic-embed-text is available and the configured chat model is unavailable, setup selects the first model for people-sweep chat, causing the chat check to fail and potentially leaving partial configuration. Require an available configured chat model or filter out embedding-only models with an actionable pending state.

Reviewers: 2 done | Synthesis: codex, 8s | Total: 14m48s

@salmonumbrella salmonumbrella mentioned this pull request Sep 3, 2026
Open
@roborev-ci

roborev-ci Bot commented Sep 3, 2026

Copy link
Copy Markdown

roborev: Combined Review (b67fa33)

Verdict: Needs changes — two medium-severity issues were identified.

Medium

  • cmd/msgvault/cmd/setup_providers.go:862-875 — Configuration changes are persisted before people-provider onboarding completes. If onboarding, consent, checking, or selection fails, setup returns an error while leaving other lanes enabled and configured, causing later runs to treat them as already configured. Stage onboarding before publishing configuration or roll back all setup edits on failure.

  • cmd/msgvault/cmd/setup_providers.go:296-309, 607-610 — The people-sweep plan adds document_text before consent decisions. If the Mistral gate is declined, the document lane is skipped but document_text remains in the OpenAI inference profile and disclosure. Recompute dependent plans after gate decisions and remove document sources when the document lane is declined.

Reviewers: 2 done | Synthesis: codex, 7s | Total: 10m44s

@wesm wesm self-assigned this Sep 5, 2026
@wesm

wesm commented Sep 5, 2026

Copy link
Copy Markdown
Member

looking

salmonumbrella and others added 2 commits September 4, 2026 19:34
@salmonumbrella @claude @wesm
feat(setup): add provider defaults and lane status
c8b95c0 
Configure unset retrieval and people lanes from available providers with
explicit consent and recommended defaults. Report which lanes are active
and the steps needed to finish setup.

- feat(setup): keep provider onboarding output consistent with the pass
- fix(setup): keep the local fallback local and disclose every source

Refs kenn-io#634

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Generated with Codex
@wesm @codex
fix(setup): restore configuration when provider onboarding fails
92f2710 
A failed people-provider check, consent, or selection left other lanes
configured, so a retry treated the incomplete setup as finished. Restore
all setup config edits on failure, including a newly created file, using
the published file identity to preserve concurrent edits.

Recompute dependent lanes after a provider is declined. Declining Mistral
must also disable document vectors and remove document_text from the
people-sweep profile and its subsequent disclosure.

Generated with Codex
Co-authored-by: Codex <noreply@openai.com>
@wesm
wesm force-pushed the feat/issue-634-smart-defaults branch from b67fa33 to 92f2710 Compare September 5, 2026 00:36
@roborev-ci

roborev-ci Bot commented Sep 5, 2026

Copy link
Copy Markdown

roborev: Combined Review (92f2710)

Verdict: Medium-severity issues remain around backend readiness, credential reporting, and sensitive-data consent.

Medium

  • cmd/msgvault/cmd/setup_providers.go:209-214, 444 — Setup selects pgvector from the database DSN without verifying that the binary includes pgvector support, potentially producing a configuration the daemon cannot start with. Check backend availability or leave the lane pending with rebuild guidance.

  • cmd/msgvault/cmd/setup_lanes.go:298-312, 340-346, 400-409 — Lane status reports visual, document, and people-sweep lanes as enabled without checking credential environment variables. Missing provider keys therefore produce no readiness warning. Report missing credentials for hosted lanes.

  • cmd/msgvault/cmd/setup_providers.go:638, 723-726 — setup providers automatically enables and consents to the people-sweep profile with allowSensitive: true, while the disclosure does not clearly state that sensitive archive excerpts may be sent and is suppressed in JSON mode. Require a separate explicit sensitive-content opt-in or disclose the policy in all modes; keep allow_sensitive false without that opt-in.

Reviewers: 2 done | Synthesis: codex, 9s | Total: 13m44s

@wesm @codex
fix(setup): require explicit consent for sensitive inference
e8eb885 
Keep the people sweep pending until --allow-sensitive explicitly permits
sensitive archive excerpts and personal inferences. Describe that policy
in both human and JSON plans; --yes alone must not authorize it.

Leave vector lanes pending when the binary lacks the database backend,
and report missing credential environment variables for hosted lanes so
the setup report does not imply that unusable providers are ready.

Fix the new documentation links to use the /docs/ URL prefix required by
the published site and its link checker.

Generated with Codex
Co-authored-by: Codex <noreply@openai.com>
@roborev-ci

roborev-ci Bot commented Sep 5, 2026

Copy link
Copy Markdown

roborev: Combined Review (e8eb885)

Summary verdict: 3 medium-severity issues require attention.

Medium

  • cmd/msgvault/cmd/setup_lanes.go:384-394: Document semantic search requires separate document_embedding and query_embedding consents, but setup only recommends msgvault documents vectors consent --yes. Track both consent purposes, disclose query-text egress, and include --purpose queries --yes.

  • cmd/msgvault/cmd/setup_providers.go:524-528, 582-586: Enabling a disabled lane overwrites existing retention and training postures with command defaults. Preserve non-empty postures and apply defaults only when unset; require an explicit override flag.

  • cmd/msgvault/cmd/setup_lanes.go:199-212, cmd/msgvault/cmd/setup_providers.go:404-424: Provider detection uses unsafe URL substring matching, allowing attacker-controlled endpoints to be classified as OpenAI or local. Parse URLs and compare exact trusted hostnames; treat other non-loopback endpoints as custom hosted providers requiring explicit handling.

Reviewers: 2 done | Synthesis: codex, 10s | Total: 32m10s

@wesm @codex
fix(setup): preserve provider policies and separate query consent
e03ba4f 
Document search sends both archive text and query text to an embedding
provider. Report their separate consent records so users can see which
permission is still missing and how to grant it.

Enabling a lane must not replace saved retention or training assertions
with command defaults. Preserve each assertion unless its own flag is
explicitly supplied.

Recognize providers by exact URL hosts, not substrings. Leave dependent
lanes on custom hosted endpoints for explicit configuration instead of
extending the data sent under another provider's disclosure.

Generated with Codex
Co-authored-by: Codex <noreply@openai.com>
@roborev-ci

roborev-ci Bot commented Sep 5, 2026

Copy link
Copy Markdown

roborev: Combined Review (e03ba4f)

Verdict: Medium-severity issues remain in provider readiness reporting, daemon onboarding, and schedule preservation.

Medium

  • cmd/msgvault/cmd/setup_lanes.go:279-291 — Lane status can report text search, people search, and document vectors as on when required provider credentials are unavailable. Mark lanes pending and propagate dependency readiness.

  • cmd/msgvault/cmd/setup_providers.go:1008; cmd/msgvault/cmd/person_provider_setup.go:1001-1029 — People-provider onboarding through an existing daemon does not forward the configured credential environment variable, causing checks to fail when the daemon lacks that environment. Forward only the required credential using the controlled environment mechanism.

  • cmd/msgvault/cmd/setup_providers.go:383-393 — Explicit empty or false schedule settings are treated as unset and overwritten with defaults. Detect explicitly present TOML keys and preserve opt-outs. The same applies to multimodal schedules.

Reviewers: 2 done | Synthesis: codex, 8s | Total: 16m14s

@wesm @codex
fix(setup): honor readiness and schedule opt-outs
f06f780 
Missing embedding credentials make text search unavailable to its people
and document consumers too. Report those lanes as pending instead of on.
Preserve explicit empty cron and false after-sync settings when applying
text and visual defaults so setup does not undo scheduling opt-outs.

A running daemon may lack the caller's provider key during onboarding.
Forward only that key for a check pinned to the saved profile fingerprint,
and preserve the caller context on the generated command. Keep ordinary
checks on daemon-owned credentials and reject keys for other profiles.

Generated with Codex
Co-authored-by: Codex <noreply@openai.com>
@roborev-ci

roborev-ci Bot commented Sep 5, 2026

Copy link
Copy Markdown

roborev: Combined Review (f06f780)

Verdict: Two medium-severity readiness and configuration issues were identified.

Medium

  • cmd/msgvault/cmd/setup_lanes.go:310-321 — Consent-gated lanes remain marked on when required consent is missing or unknown, despite runtime gates preventing operation. Set the lane state to pending in this case.

  • cmd/msgvault/cmd/setup_providers.go:489-500 — Local Ollama setup changes the embedding endpoint and format but leaves an existing api_key_env set, potentially causing a disabled profile to remain permanently pending due to stale credentials. Clear api_key_env for unauthenticated local Ollama, or skip automatic selection when it was explicitly configured.

Reviewers: 2 done | Synthesis: codex, 9s | Total: 29m7s

@wesm @codex
fix(setup): keep unconsented lanes pending
29dffae 
Enabled configuration does not make a consent-gated lane ready. Report
missing or unreadable consent as pending, and keep document vectors
pending until both document and query consent are active.

Local Ollama selection uses an unauthenticated endpoint. Clear the old
embedding credential reference so an unused hosted key cannot leave the
new local lane pending.

Generated with Codex
Co-authored-by: Codex <noreply@openai.com>
@roborev-ci

roborev-ci Bot commented Sep 5, 2026

Copy link
Copy Markdown

roborev: Combined Review (29dffae)

Verdict: Two medium-severity issues require attention; no high or critical findings were reported.

Medium

  • cmd/msgvault/cmd/setup_lanes.go:279-289 — Configured vector lanes are reported as on without verifying that the selected SQLite/pgvector backend is compiled into the binary. This can cause daemon startup failure and contradict documented pending/rebuild behavior. Check backend availability and provide rebuild guidance when unavailable.

  • cmd/msgvault/cmd/setup_providers.go:631 — Document-vector follow-ups only include consent commands, which do not build the vector index. Add msgvault documents vectors build after both consent commands so semantic search is usable.

Reviewers: 2 done | Synthesis: codex, 9s | Total: 18m29s

@wesm @codex
fix(setup): report unavailable vector backends and finish indexing
ed3ac6e 
Configured vector lanes cannot run when the binary lacks their database
backend. Use the same compiled-backend check for setup and status, and
carry the pending state to dependent lanes with rebuild guidance.

Consent records authorize document-vector work but do not create an
index. Include the build command after both consent steps so the setup
follow-ups finish the indexing work.

Generated with Codex
Co-authored-by: Codex <noreply@openai.com>
@roborev-ci

roborev-ci Bot commented Sep 5, 2026

Copy link
Copy Markdown

roborev: Combined Review (ed3ac6e)

Verdict: Two medium-severity issues require attention; no critical or high-severity findings were reported.

Medium

  • cmd/msgvault/cmd/setup_lanes.go:214-218 — Visual consent is reported as active whenever any active/building generation is consented, without matching the current multimodal configuration or capability-manifest policy. After configuration changes, runtime consent is rejected while setup status still reports the lane as on. Require generation and consent-policy fingerprints to match; otherwise report pending with the rebuild/consent command.

  • cmd/msgvault/cmd/setup_providers.go:874-877, 939-943 — The literal unknown posture is treated as an explicit assertion. Enabling documents with the documented unknown defaults creates an enabled lane that documents consent-mistral refuses, while later setup runs treat it as already enabled and do not apply posture flags. Treat unknown as unset or keep the lane pending until explicit postures are supplied, and allow posture flags to repair an enabled unknown configuration.

Reviewers: 2 done | Synthesis: codex, 10s | Total: 29m20s

@wesm @codex
fix(setup): match visual consent and complete unknown postures
6f98633 
A consented visual generation may belong to an old configuration or
capability manifest. Compare both fingerprints before reporting active
consent, using the same upload policy calculation as the runtime.

The document commands reject unknown postures, so retaining those values
leaves setup unable to finish. Treat unknown as unset and let the Mistral
confirmation complete those assertions even on an enabled document lane.
Preserve known assertions unless their corresponding flags replace them.

Generated with Codex
Co-authored-by: Codex <noreply@openai.com>
@roborev-ci

roborev-ci Bot commented Sep 5, 2026

Copy link
Copy Markdown

roborev: Combined Review (6f98633)

Verdict: 3 medium-severity issues require attention; no critical or high-severity issues found.

Medium

  • cmd/msgvault/cmd/setup_providers.go:216-219, 543-548 — Setup treats any existing manifest as valid without decoding or validating required capabilities such as CapabilityQueryText, potentially enabling a lane that runtime validation rejects.

  • cmd/msgvault/cmd/setup_lanes.go:392-424, cmd/msgvault/cmd/setup_providers.go:216-219 — An explicitly configured but missing capabilities_file is ignored, allowing setup to use or overwrite the default path instead of the configured custom path.

  • cmd/msgvault/cmd/setup_lanes.go:211-213 — Document consent is reported from any active document-provider profile rather than the currently configured provider, model, region, and postures, which can produce stale active status after configuration changes.

Reviewers: 2 done | Synthesis: codex, 9s | Total: 29m7s

@wesm @codex
fix(setup): validate manifests and match document provider consent
4c57c09 
An existing manifest does not make visual search usable. Apply the runtime
policy checks before enabling the lane, including text-query authority,
and report invalid manifests as pending. Keep an explicit manifest path
even when it is missing so probe guidance respects the operator's choice.

Document consent from an older provider configuration must not make the
current lane appear ready. Match the configured provider, endpoint, region,
model, and postures against active consented profiles without requiring
the original probe file to remain at setup's default location.

Generated with Codex
Co-authored-by: Codex <noreply@openai.com>
@roborev-ci

roborev-ci Bot commented Sep 5, 2026

Copy link
Copy Markdown

roborev: Combined Review (4c57c09)

Verdict: Two medium-severity issues require attention.

Medium

  • cmd/msgvault/cmd/setup_lanes.go:612-623
    liveMCPTools infers tool availability solely from configuration flags. If vector initialization fails, setup status still reports those tools as live. Derive this list from actual daemon capability/readiness state, or label it as configured tools.

  • cmd/msgvault/cmd/setup_lanes.go:231-239, internal/store/document_index.go:480-498
    Document consent matching ignores the immutable profile fingerprint and policy JSON, comparing only selected fields. Configuration changes to limits, normalization, scope, or allowed media can leave older consent reported as active. Compare the complete extraction policy, or report consent as pending when an exact match cannot be established.

Reviewers: 2 done | Synthesis: codex, 9s | Total: 22m52s

@wesm @codex
test(setup): preserve configured path spelling on Windows
c46dbd1 
The custom-manifest fixture writes forward slashes into TOML on every
platform. Setup preserves that spelling, but the test expected native
Windows separators and failed the Windows CLI shard. Match the fixture's
path spelling while retaining the checks that setup preserves the custom
path and uses it in probe guidance.

Generated with Codex
Co-authored-by: Codex <noreply@openai.com>
@roborev-ci

roborev-ci Bot commented Sep 5, 2026

Copy link
Copy Markdown

roborev: Combined Review (c46dbd1)

Verdict: 3 actionable findings identified—1 High and 2 Medium.

High

  • cmd/msgvault/cmd/person_provider_setup.go:1041-1048 — Provider credentials are forwarded to remote daemons via CLIRunRequest.Env, exposing API keys to observers of allowed remote connections or compromised daemon hosts. Require validated HTTPS transport or perform the check locally and send only its result.

Medium

  • cmd/msgvault/cmd/setup_lanes.go:177-187, cmd/msgvault/cmd/setup_providers.go:535-537 — When multimodal search is enabled with an empty capabilities_file, the computed default manifest path is not applied to validation. Setup reports the lane as pending even when the default manifest exists. Resolve and use the default path, persisting it when appropriate.

  • internal/store/document_index.go:480-498 — Document consent matching ignores the profile fingerprint and content-policy fields. Configuration changes can leave setup status reporting active consent while document builds reject the configuration. Match the complete immutable policy identity or report consent as missing when it cannot be verified.

Reviewers: 2 done | Synthesis: codex, 10s | Total: 27m41s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@wesm wesm

Labels

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@salmonumbrella @wesm